{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# 2.1 Building a new employee Q&A robot with an LLM\n",
        "\n",
        "## 🚄 Preface\n",
        "You work at an educational content development company, and the continuous influx of new employees has led to frequent questions, resulting in significant time and resource costs. To address this, you’ve decided to leverage LLM technology to build an AI-powered Q&A bot that improves response accuracy and efficiency.\n",
        "\n",
        "## 🍁 Goals\n",
        "After completing this course, you will be able to:\n",
        "\n",
        "* Call Qwen-Max through its API\n",
        "* Understand how LLMs work\n",
        "* Recognize the limitations of LLMs and apply practical solutions\n",
        "\n",
        "## ⚠️ Environment preparation\n",
        "To ensure a smoother experience with the tutorial, we recommend that you first complete the [**Environment Preparation**](https://edu.aliyun.com/course/3130200/lesson/343310285) chapter in the **Alibaba Cloud Large Language Model Senior Engineer ACP Certification Course** (alibabacloud_acp_learning), and ensure that the required environment for the course is correctly installed.\n",
        "\n",
        "To ensure a smooth experience with this tutorial, we recommend completing the the [**Environment Preparation**](https://edu.aliyun.com/course/3130200/lesson/343310285) chapter in the **Alibaba Cloud Large Language Model Senior Engineer ACP Certification Course**  (alibabacloud_acp_learning) and verifying that all required dependencies are properly installed.\n",
        "\n",
        "\n",
        "\n",
        "## 💻 1. Calling Qwen-Max via API\n",
        "\n",
        "The most direct way to interact with an LLM is through a web interface (such as [Qwen-Max](https://tongyi.aliyun.com/qianwen/)). However, as a developer,  you’ll often need to integrate LLM capabilities into your own applications. You can use the widely adopted OpenAI-compatible Python SDK to call the Qwen-Max. You have already installed the necessary dependencies in `1.0 Computing Environment Setup`. Before running the code below, ensure you've activated the correct Python environment, such as the `Python (llm_learn)` environment newly created in this tutorial.\n",
        "\n",
        "<a href=\"https://img.alicdn.com/imgextra/i3/O1CN01ytNXS81RGWTNtmph7_!!6000000002084-2-tps-3448-1596.png\" target=\"_blank\">\n",
        "<img src=\"https://img.alicdn.com/imgextra/i3/O1CN01ytNXS81RGWTNtmph7_!!6000000002084-2-tps-3448-1596.png\" width=\"800\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://img.alicdn.com/imgextra/i1/O1CN01B9bNMT27MDFvpBmnc_!!6000000007782-2-tps-1944-448.png\" target=\"_blank\">\n",
        "<img src=\"https://img.alicdn.com/imgextra/i1/O1CN01B9bNMT27MDFvpBmnc_!!6000000007782-2-tps-1944-448.png\" width=\"800\">\n",
        "</a>  \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "To invoke Qwen, go to Model Studio, Alibaba Cloud's LLM service platform, and activate [Model Invocation Service](https://bailian.console.aliyun.com/#/model-market), then [Create an API key](https://bailian.console.aliyun.com/?apiKey=1#/api-key).\n",
        "\n",
        "> If the following message appears at the top of the page, it means you haven’t activated the Model Studio invocation service yet. Once activated, you’ll be able to invoke the model.\n",
        "> <img src=\"https://img.alicdn.com/imgextra/i4/O1CN017dWRx21XtLkBg4rMZ_!!6000000002981-2-tps-1236-254.png\" width=\"600\">\n",
        "\n",
        "Before using the API, you must securely manage your API key. Hardcoding the key directly in your source code is a security risk—it can easily be exposed when sharing code, and any plaintext references must be updated manually whenever the key is rotated. A safer and more maintainable approach is to store the API key in an environment variable.\n",
        "\n",
        "The following code loads your API key from a configuration file and sets it as an environment variable. After execution, it displays the first five characters of the key followed by asterisks, allowing you to verify the configuration without exposing the full key."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-11-15T08:58:03.667571Z",
          "iopub.status.busy": "2024-11-15T08:58:03.667184Z",
          "iopub.status.idle": "2024-11-15T08:59:44.060371Z",
          "shell.execute_reply": "2024-11-15T08:59:44.059650Z",
          "shell.execute_reply.started": "2024-11-15T08:58:03.667545Z"
        },
        "tags": []
      },
      "outputs": [],
      "source": [
        "# Load the API Key for invoking the Qwen LLM\n",
        "import os\n",
        "from config.load_key import load_key\n",
        "load_key()\n",
        "print(f'Your configured API Key is: {os.environ[\"DASHSCOPE_API_KEY\"][:5]+\"*\"*5}')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "You need to press “Enter” to confirm the “API-KEY” you entered. After successful input, you'll see the message ```“The API Key you configured is: sk-88***** ”```. \n",
        "If you need to change the “API-KEY”, please edit the “KEY.json” file in the parent directory. \n",
        "If you are using VS-CODE, the input box for the API-KEY will appear at the **top** of the window.  \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's start with a simple conversation. The following code creates an assistant named \"Company Assistant,\" that answers questions about company operations. Try it with the example question: \"Choosing a project management tool\"\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-11-15T08:59:46.180449Z",
          "iopub.status.busy": "2024-11-15T08:59:46.179984Z",
          "iopub.status.idle": "2024-11-15T08:59:59.498380Z",
          "shell.execute_reply": "2024-11-15T08:59:59.497544Z",
          "shell.execute_reply.started": "2024-11-15T08:59:46.180423Z"
        },
        "tags": []
      },
      "outputs": [],
      "source": [
        "from openai import OpenAI\n",
        "import os\n",
        "\n",
        "client = OpenAI(\n",
        "    api_key=os.getenv(\"DASHSCOPE_API_KEY\"),\n",
        "    base_url=\"https://dashscope.aliyuncs.com/compatible-mode/v1\",\n",
        ")\n",
        "\n",
        "def get_qwen_response(prompt):\n",
        "    response = client.chat.completions.create(\n",
        "        model=\"qwen-plus-0919\",\n",
        "        messages=[\n",
        "            # The system message is used to set the role and task of the LLM\n",
        "            {\"role\": \"system\", \"content\": \"You are an assistant responsible for answering questions in a global educational content development company. Your name is Alex, and you're here to help your colleagues with their inquiries.\"},\n",
        "            # The user message is used to input the user's question\n",
        "            {\"role\": \"user\", \"content\": prompt}\n",
        "        ]\n",
        "    )\n",
        "    return response.choices[0].message.content\n",
        "\n",
        "response = get_qwen_response(\"What tools should our company use for project management?\")\n",
        "print(response)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "To implement multi-turn conversations—where the LLM references previous dialogue history to generate context-aware replies—see the [multi-round conversation](https://help.aliyun.com/zh/model-studio/user-guide/text-generation#865b38621dwin).\n",
        "\n",
        "After running the code above, you'll notice that it takes some time (about 20 seconds) to receive the complete response. This happens because, by default, the API waits until the model finishes generating the full response before returning it all at once. In real-world applications, this delay can negatively impact user experience—imagine users staring at a blank screen for 20 seconds!\n",
        "\n",
        "Fortunately, you can use \"streaming output\" to address this issue. With streaming output, the model sends responses incrementally as it generates them, much like how humans type, so users can see partial results immediately. This significantly improves the interactive experience. Next, let’s see how to implement streaming output...\n",
        "> 💡 Tip: Streaming output only changes the way content is displayed; the model's reasoning process and the quality of the final answer remain unchanged. You can confidently use this feature to improve your application's user experience.\n",
        "\n",
        "To implement streaming output, simply add the `stream=True` parameter to the previous code and adjust the output method:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-11-15T09:00:04.752657Z",
          "iopub.status.busy": "2024-11-15T09:00:04.752276Z",
          "iopub.status.idle": "2024-11-15T09:00:18.769987Z",
          "shell.execute_reply": "2024-11-15T09:00:18.769374Z",
          "shell.execute_reply.started": "2024-11-15T09:00:04.752633Z"
        },
        "tags": []
      },
      "outputs": [],
      "source": [
        "def get_qwen_stream_response(user_prompt, system_prompt):\n",
        "    response = client.chat.completions.create(\n",
        "        model=\"qwen-plus-0919\",\n",
        "        messages=[\n",
        "            {\"role\": \"system\", \"content\": system_prompt},\n",
        "            {\"role\": \"user\", \"content\": user_prompt}\n",
        "        ],\n",
        "        stream=True\n",
        "    )\n",
        "    for chunk in response:\n",
        "        yield chunk.choices[0].delta.content\n",
        "\n",
        "response = get_qwen_stream_response(user_prompt=\"What tools should our company use for project management?\", system_prompt=\"You are an assistant responsible for answering questions in a global educational content development company. Your name is Alex, and you're here to help your colleagues with their inquiries.\")\n",
        "for chunk in response:\n",
        "    print(chunk, end=\"\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "By querying the Qwen-Max model twice, you might notice some interesting phenomena:\n",
        "1. **Even with identical questions**, the responses may vary slightly each time. This is one of the inherent characteristics of LLMs. Much like humans, LLMs can express similar ideas in different ways.\n",
        "2. **The suggestions provided by the model** tend to focus on commonly used project management tools such as Jira, Trello, or Asana. While LLMs possess broad knowledge, they don't have access to your company's specific context — for example, your current toolchain, team size, or budget constraints.\n",
        "> 💡 You can find information about the project management software currently used in your company in the file docs/Content_Developer_Job_Guide.pdf.\n",
        "\n",
        "These two observations are actually quite fascinating! Why do LLMs behave this way? To understand this, we need to lift the \"mysterious veil\" surrounding LLMs and explore how they think and make decisions. Don’t worry — you'll get a clear and intuitive explanation without any complex technical jargon.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 📚 2. How LLMs work\n",
        "In recent decades, artificial intelligence (AI) has experienced a transformative journey from rudimentary algorithms to the advent of generative AI. This advanced form of AI can produce entirely new content—spanning text, images, audio, and video—by learning from extensive datasets. Such capabilities have significantly propelled the proliferation of AI across myriad applications. Common use cases include intelligent question-answering systems (like Qwen-Max and GPT), creative image generation (such as with Stable Diffusion), and code synthesis tools (like Lingma), making AI more accessible and applicable across various sectors.\n",
        "\n",
        "<a href=\"https://img.alicdn.com/imgextra/i3/O1CN01AahaFq1O5aBWfg4Ur_!!6000000001654-2-tps-3424-1342.png\" target=\"_blank\">\n",
        "<img src=\"https://img.alicdn.com/imgextra/i3/O1CN01AahaFq1O5aBWfg4Ur_!!6000000001654-2-tps-3424-1342.png\" width=\"1000\">\n",
        "</a>\n",
        "\n",
        "Intelligent question answering represents one of the most iconic and extensively utilized applications of LLMs, serving as an exemplary model for understanding how LLMs operate. Below, we'll delve into the workflow of LLMs within Q&A scenarios to offer deeper insights into the underlying technical mechanisms.\n",
        "\n",
        "### 2.1. The Q&A workflow of LLMs\n",
        "Below is an example where the input text “ACP is a very” initiates a query to the LLM. The diagram below shows the complete process from initiating the query to outputting the text.\n",
        "\n",
        "<a href=\"https://img.alicdn.com/imgextra/i2/O1CN01TR39fC1xOMz2zEvlu_!!6000000006433-2-tps-3432-1866.png\" target=\"_blank\">\n",
        "<img src=\"https://img.alicdn.com/imgextra/i2/O1CN01TR39fC1xOMz2zEvlu_!!6000000006433-2-tps-3432-1866.png\" width=\"1000\">\n",
        "</a>\n",
        "\n",
        "The Q&A workflow of LLMs consists of five stages:\n",
        "\n",
        "**Stage 1: Tokenization of input text**\n",
        "\n",
        "Token is the basic unit of text processing in LLMs, typically representing words, phrases, or symbols. We need to break down the sentence “ACP is a very” into smaller units with independent semantic meaning (tokens) and assign each token an ID. If necessary, please reference [Tokenization](https://github.com/QwenLM/Qwen/blob/main/tokenization_note.md).\n",
        "\n",
        "<a href=\"https://gw.alicdn.com/imgextra/i1/O1CN019gAS3k1DrhwpIdHl6_!!6000000000270-0-tps-2414-546.jpg\" target=\"_blank\">\n",
        "<img src=\"https://gw.alicdn.com/imgextra/i1/O1CN019gAS3k1DrhwpIdHl6_!!6000000000270-0-tps-2414-546.jpg\" width=\"1000\">\n",
        "</a>\n",
        "\n",
        "**Stage 2: Token embedding**\n",
        "\n",
        "Computers can only understand numbers and cannot directly comprehend the meaning of tokens. Therefore, tokens must be converted into numerical representations (i.e., vectors) so that they can be understood by computers. Token embedding transforms each token into a vector of fixed dimensions.\n",
        "\n",
        "\n",
        "**Stage 3: Inference by the LLM**\n",
        "\n",
        "The LLM learns knowledge from vast amounts of pre-trained data. When we input new content, such as “ACP is a very,” the LLM combines its learned knowledge to make predictions. It calculates the probabilities of all possible tokens and generates a set of candidate token. Finally, the LLM selects one token as the next output based on these calculations.\n",
        "\n",
        "This explains why, when asked about internal project management tools within a company, the model cannot provide suggestions for internal tools, as its predictive ability is based solely on the pre-trained data and it lacks knowledge of information it has not been exposed to. Therefore, when requiring a Q&A bot to answer domain-specific questions, this issue needs to be addressed specifically, which will be further elaborated in Section 3 of this chapter.\n",
        "\n",
        "**Stage 4: Output tokens**\n",
        "\n",
        "Since the LLM randomly selects from the candidate tokens based on their probabilities, this leads to the phenomenon that “even if the question is exactly the same, the answers are slightly different each time.” To control the randomness of the generated content, parameters such as temperature and top-p are commonly adjusted.\n",
        "\n",
        "For example, in the figure below, the first set of candidate tokens output by the LLM is “informative (50%),” “fun (25%),” “enlightening (20%),” and “boring (5%).” Adjusting the temperature or top-p parameters will influence the LLM's preference in selecting from the candidate tokens, such as choosing the highest probability option, “informative.” You can learn more about these two parameters in Section 2.2 of this chapter.\n",
        "\n",
        "| Candidate Token | Probability |\n",
        "|-----------------|-------------|\n",
        "| informative     | 50%         |\n",
        "| fun             | 25%         |\n",
        "| enlightening    | 20%         |\n",
        "| boring          | 5%          |\n",
        "\n",
        "Specifically, “informative” will continue to be fed back into the LLM to generate subsequent candidate tokens. This process is called an autoregressive model, utilizing both the input text and previously generated text. The LLM uses this method to sequentially generate candidate tokens.\n",
        "\n",
        "**Stage 5: Output text**\n",
        "\n",
        "The processes of Stage 3 and Stage 4 are repeated until a special token (such as <EOS>, end-of-sentence) is output or the output length reaches a threshold, concluding the Q&A session. The LLM then outputs all generated content. Of course, you can utilize the streaming output capability of the LLM, which predicts and immediately returns some tokens. In this example, the final output would be “ACP is a very informative course.”"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 2.2 Parameters affecting the randomness of content generation in LLMs\n",
        "\n",
        "Assume a question-and-answer scenario where the user asks: \"What can you learn in the ACP course on LLMs?\" To simulate the content generation process of an LLM, we have pre-defined a candidate token set consisting of the following tokens: \"**RAG**\", \"**prompt**\", \"**model**\", \"**writing**\", and \"**drawing**\". The LLM will select one of these five candidate tokens as the output (next-token), as shown below.\n",
        "> User question: What can you learn in the ACP course?<br><br>\n",
        "> LLM response: RAG <br>\n",
        "\n",
        "In this process, two important parameters influence the LLM's output: temperature and top_p. These parameters control the randomness and diversity of the content generated by the LLM. Below, we introduce how these two parameters work and how to use them.\n",
        "\n",
        "#### 2.2.1 Temperature: Adjusting the probability distribution of the candidate token set\n",
        "\n",
        "Before generating the next token, the LLM first calculates an initial probability distribution for the candidate tokens. This distribution represents the probability of each candidate token being selected as the next-token. Temperature acts as a regulator that alters the probability distribution of the candidate tokens, influencing the content generation of the LLM. By adjusting this parameter, you can flexibly control the diversity and creativity of the generated text.\n",
        "\n",
        "To better understand, the figure below illustrates the impact of different temperature values on the probability distribution of the candidate tokens. The plotting code is located in the /resources/2_1 directory.\n",
        "\n",
        "<a href=\"https://img.alicdn.com/imgextra/i2/O1CN01kMbKhx1cLyzCyfRQW_!!6000000003585-2-tps-1728-600.png\" target=\"_blank\">\n",
        "<img src=\"https://img.alicdn.com/imgextra/i2/O1CN01kMbKhx1cLyzCyfRQW_!!6000000003585-2-tps-1728-600.png\" width=\"1000\">\n",
        "</a>\n",
        "\n",
        "The low, medium, and high temperatures settings in the figure are based on the range [0, 2) of the Qwen-Plus model.\n",
        "\n",
        "As shown in the figure, as the temperature increases from low to high (0.1 -> 0.7 -> 1.2), the probability distribution becomes flatter. The probability of the candidate token \"RAG\" decreases from 0.8 -> 0.6 -> 0.3. Although it remains the most probable token, its probability is now closer to those of other candidate tokens. Consequently, the final output transitions from relatively fixed to increasingly diverse.\n",
        "\n",
        "Depending on the use case, consider the following recommendations for setting the temperature parameter:\n",
        "- Clear answers (e.g., generating code): Lower the temperature.\n",
        "- Creative variety (e.g., ad copy): Increase the temperature.\n",
        "- No special requirements: Use the default temperature (usually within the medium range).\n",
        "\n",
        "Note that when temperature = 0, although randomness is minimized, it does not guarantee identical outputs every time. For a deeper understanding, you can refer to [the underlying algorithm implementation of temperature](https://github.com/huggingface/transformers/blob/v4.49.0/src/transformers/generation/logits_process.py#L226).\n",
        "\n",
        "Next, let’s experience the effect of temperature. By adjusting the temperature value, ask the same question 10 times and observe the fluctuations in the responses.\n",
        "> The example code for temperature is similar to the one uesd in the upcoming explanation of top_p, so it has been encapsulated for subsequent use."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "ExecutionIndicator": {
          "show": true
        },
        "execution": {
          "iopub.execute_input": "2024-11-15T12:06:39.470715Z",
          "iopub.status.busy": "2024-11-15T12:06:39.470351Z",
          "iopub.status.idle": "2024-11-15T12:06:44.434877Z",
          "shell.execute_reply": "2024-11-15T12:06:44.434229Z",
          "shell.execute_reply.started": "2024-11-15T12:06:39.470689Z"
        },
        "tags": []
      },
      "outputs": [],
      "source": [
        "import time\n",
        "\n",
        "def get_qwen_stream_response(user_prompt, system_prompt, temperature, top_p):\n",
        "    response = client.chat.completions.create(\n",
        "        model=\"qwen-max\",\n",
        "        messages=[\n",
        "            {\"role\": \"system\", \"content\": system_prompt},\n",
        "            {\"role\": \"user\", \"content\": user_prompt}\n",
        "        ],\n",
        "        temperature=temperature,\n",
        "        top_p=top_p,\n",
        "        stream=True\n",
        "    )\n",
        "\n",
        "    for chunk in response:\n",
        "        yield chunk.choices[0].delta.content\n",
        "\n",
        "\n",
        "# Prints streaming output from Qwen-Plus with optional parameters\n",
        "def print_qwen_stream_response(\n",
        "    user_prompt,\n",
        "    system_prompt,\n",
        "    temperature=0.7,\n",
        "    top_p=0.8,\n",
        "    iterations=10\n",
        "):\n",
        "    for i in range(iterations):\n",
        "        print(f\"Output {i + 1}: \", end=\"\")\n",
        "        # Delay to avoid hitting API rate limits\n",
        "        time.sleep(0.5)\n",
        "        response = get_qwen_stream_response(user_prompt, system_prompt, temperature, top_p)\n",
        "        output_content = ''\n",
        "        for chunk in response:\n",
        "            if chunk:  # Avoid NoneType errors\n",
        "                output_content += chunk\n",
        "        print(output_content)\n",
        "\n",
        "\n",
        "# Qwen-Plus Model Info:\n",
        "# - Temperature range: [0, 2), default: 0.7\n",
        "# - Set temperature=0 for deterministic output\n",
        "print_qwen_stream_response(\n",
        "    user_prompt=\"A horse can also be referred to as\",\n",
        "    system_prompt=\"Continue this sentence with a phrase of five words or fewer.\",\n",
        "    temperature=0\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-11-15T12:06:45.417531Z",
          "iopub.status.busy": "2024-11-15T12:06:45.417146Z",
          "iopub.status.idle": "2024-11-15T12:06:50.629069Z",
          "shell.execute_reply": "2024-11-15T12:06:50.628320Z",
          "shell.execute_reply.started": "2024-11-15T12:06:45.417506Z"
        },
        "tags": []
      },
      "outputs": [],
      "source": [
        "# Set temperature=1.9\n",
        "print_qwen_stream_response(user_prompt=\"A horse can also be referred to as\", system_prompt=\"Continue this sentence with a phrase of five words or fewer.\", temperature=1.9)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "jupyter": {
          "outputs_hidden": true
        },
        "tags": []
      },
      "source": [
        "The experiment clearly shows that as the temperature increases, the model generates increasingly varied and diverse content.\n",
        "\n",
        "#### 2.2.2 top_p: Control the sampling range of the candidate token set\n",
        "\n",
        "top_p (also known as nucleus sampling) is a filtering mechanism that selects a subset of candidate tokens whose cumulative probability meets a specified threshold. Specifically, tokens are sorted in descending order of probability, and the smallest set of top-ranked tokens whose cumulative probability is **at least** the top-p threshold is selected as the new candidate set.\n",
        "\n",
        "The figure below shows the sampling effect of different top_p values on the candidate token set.\n",
        "\n",
        "<a href=\"https://img.alicdn.com/imgextra/i1/O1CN01aRiNYq1ekpiIU6Q89_!!6000000003910-2-tps-3080-1188.png\" target=\"_blank\">\n",
        "<img src=\"https://img.alicdn.com/imgextra/i1/O1CN01aRiNYq1ekpiIU6Q89_!!6000000003910-2-tps-3080-1188.png\" width=\"1000\">\n",
        "</a>\n",
        "\n",
        "In the illustration, the blue part represents tokens whose cumulative probability reaches the top_p threshold (e.g., 0.5 or 0.8), forming a new candidate set; the gray part represents tokens that are not selected.\n",
        "\n",
        "When top_p = 0.5, only the highest-probability token (“RAG”) is included in the sampling pool; when top_p = 0.8, the pool expands to include “RAG,” “prompt,” and “model,” from which the next token is sampled randomly.\n",
        "\n",
        "In summary, the effect of the top_p parameter on LLM output can be described as follows:\n",
        "- Higher top_p values: Wider candidate range, more diverse content, ideal for tasks like poetry, storytelling, or brainstorming.\n",
        "- Lower top_p values: Narrower candidate range, more stable output, suitable for news drafts, code generation, and other scenarios requiring clear answers.\n",
        "- Extremely low top_p values (e.g., 0.0001): Theoretically, the model only selects the highest-probability token, resulting in very stable output. However, in practice, due to factors such as distributed systems and additional adjustments to model outputs, slight randomness may still be introduced, making it impossible to guarantee completely consistent output every time.\n",
        "\n",
        "\n",
        "Next, explore the effect of top_p by running the same question 10 times with different top_p values and observing the variation in responses.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-11-15T12:06:52.894507Z",
          "iopub.status.busy": "2024-11-15T12:06:52.894139Z",
          "iopub.status.idle": "2024-11-15T12:06:58.171739Z",
          "shell.execute_reply": "2024-11-15T12:06:58.171137Z",
          "shell.execute_reply.started": "2024-11-15T12:06:52.894481Z"
        },
        "tags": []
      },
      "outputs": [],
      "source": [
        "# Qwen-Plus model: The value range of top_p is (0,1], with a default value of 0.8.\n",
        "# Set top_p=0.001\n",
        "print_qwen_stream_response(\n",
        "    user_prompt=\"Name an intelligent gaming smartphone, it could be\",\n",
        "    system_prompt=\"Suggest a short and catchy name for a smart gaming smartphone, using 4 words or fewer.\",\n",
        "    top_p=0.001\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-11-15T12:06:58.394593Z",
          "iopub.status.busy": "2024-11-15T12:06:58.394205Z",
          "iopub.status.idle": "2024-11-15T12:07:03.378361Z",
          "shell.execute_reply": "2024-11-15T12:07:03.377556Z",
          "shell.execute_reply.started": "2024-11-15T12:06:58.394567Z"
        },
        "tags": []
      },
      "outputs": [],
      "source": [
        "# Set top_p=0.8\n",
        "print_qwen_stream_response(\n",
        "    user_prompt=\"Name an intelligent gaming smartphone, it could be\", \n",
        "    system_prompt=\"Suggest a short and catchy name for a smart gaming smartphone, using 4 words or fewer.\", \n",
        "    top_p=0.8\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Based on the experimental results, it can be observed that the higher the top_p value, the greater the randomness in the output of LLMs."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### 2.2.3 Summary\n",
        "\n",
        "**Should temperature and top_p be adjusted simultaneously?**\n",
        "\n",
        "While temperature and top_p influence randomness in different ways, they are commonly used together in practice. However, for beginners or during initial experimentation, it’s advisable to adjust one parameter at a time to better understand its individual effect. Once familiar with their behavior, combining both allows finer control over creativity and coherence.\n",
        "\n",
        "><br>**Knowledge Extension: top_k**<br>In the Qwen series models, the parameter top_k also has capabilities similar to top_p. Refer to the [Qwen API Documentation](https://help.aliyun.com/zh/model-studio/developer-reference/use-qwen-by-calling-api?spm=a2c4g.11186623.help-menu-2400256.d_3_3_0.68332bdb2Afk2s&scm=20140722.H_2712576._.OR_help-V_1). Top_k is a sampling method that restricts the next token selection to the k most probable tokens, from which one is sampled according to the (optionally temperature-scaled) probabilities. Generally speaking, the larger the top_k, the more diverse the generated content; the smaller the top_k, the more fixed the content. When top_k is set to 1, the model only selects the Token with the highest probability, making the output more stable but also causing a lack of variation and creativity.<br><br>\n",
        ">**Knowledge Extension: seed**<br>In the Qwen series models, the parameter seed also supports controlling the determinism of the generated content. Refer to the [Qwen API Documentation](https://help.aliyun.com/zh/model-studio/developer-reference/use-qwen-by-calling-api?spm=a2c4g.11186623.help-menu-2400256.d_3_3_0.68332bdb2Afk2s&scm=20140722.H_2712576._.OR_help-V_1). The seed parameter enables reproducible sampling. When the same seed, prompt, and generation parameters (e.g., temperature, top_p) are used, the model will typically produce identical outputs. However, full determinism is not always guaranteed due to factors such as hardware-level non-determinism (e.g., GPU floating-point operations) or backend implementation details.\n",
        "<br><br>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "**Why does randomness still exist when controlling the output of LLMs (LLM) by setting temperature, top_p, and seed?**\n",
        "\n",
        "Even if temperature is set to 0, top_p is set to an extremely small value (e.g., 0.0001), and the same seed is used, the generated results for the same question may still be inconsistent. This is because some complex factors may introduce slight randomness, such as the LLMs running in a distributed system or optimization being applied to the model's output.\n",
        "\n",
        "For example:\n",
        "A distributed system is like slicing bread with different machines. Although each machine operates according to the same settings, subtle differences between devices may still result in slightly different slices of bread.  \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## ⚙️ 3. Enable LLMs to answer private knowledge questions\n",
        "To enable LLMs to answer private knowledge questions, you can choose one of the following two approaches:\n",
        "- **Prompt Injection (No Model Changes)**<br>\n",
        "    Provide the relevant internal knowledge directly within the prompt when making a request.\n",
        "- **Model Customization (Fine-tuning or Training)**<br>\n",
        "    Fine-tune or train a new model to better understand and respond to domain-specific queries.\n",
        "\n",
        "Given the high cost and complexity of model training, for most internal knowledge Q&A use cases, it's often more efficient to include the necessary context directly in the prompt."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# User's question\n",
        "user_question = \"I'm part of Software Team 1. What project management tool should we be using?\"\n",
        "\n",
        "# Internal knowledge about company tools\n",
        "knowledge = \"\"\"The company provides two main project management tools:\n",
        "  \n",
        "  1. **Jira**: Best suited for software development teams practicing agile methodologies like Scrum and Kanban. It offers robust features such as issue tracking, sprint planning, and time logging.\n",
        "  \n",
        "  2. **Microsoft Project**: Designed for enterprise-level or complex projects requiring detailed planning, resource allocation, and budget control. This is ideal for teams needing strict oversight over timelines and costs.\n",
        "  \n",
        "  Recommendation: While Microsoft Project is the company-standard solution and fully licensed, Software Teams 1, 3, and 4 are currently using Jira. There are plans to transition these teams to Microsoft Project by mid-2026.\n",
        "\"\"\"\n",
        "\n",
        "# Use a named assistant persona (for clarity and branding)\n",
        "response = get_qwen_stream_response(\n",
        "    user_prompt=user_question,\n",
        "    system_prompt=\"You are Alex, an internal support assistant at an educational content development company. Use the provided context to answer the user’s question clearly and accurately.\" + knowledge,\n",
        "    temperature=0.7,\n",
        "    top_p=0.8\n",
        ")\n",
        "\n",
        "for chunk in response:\n",
        "    print(chunk, end=\"\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "When you provide reference information directly within a prompt, LLMs can answer questions involving private or internal knowledge. However, this approach has notable limitations. Prompts have maximum length constraints, and including large amounts of background data can lead to overly long inputs. This not only reduces model efficiency but may also hit token limits, preventing the model from generating a response altogether.\n",
        "\n",
        "To address these limitations, you can implement a system that automatically retrieves relevant internal knowledge when a user asks a question. The retrieved document snippets are then combined with the user’s query and fed into the LLM to generate a final answer. This way, you avoid flooding the prompt with excessive background data. This approach is commonly referred to as **Retrieval-Augmented Generation (RAG)**.\n",
        "\n",
        "A typical RAG pipeline consists of two main stages:\n",
        "\n",
        "### 3.1 Indexing phase\n",
        "<img src=\"https://img.alicdn.com/imgextra/i3/O1CN01h0y0Uy1WH30Q7FRDJ_!!6000000002762-2-tps-1592-503.png\" width=\"1000\">\n",
        "\n",
        "During the indexing phase, internal knowledge documents or fragments are transformed into a format suitable for fast and accurate retrieval. This involves splitting the content into chunks, converting each chunk into a numerical representation (called an embedding) using specialized models, and storing these embeddings in a vector database.\n",
        "\n",
        "This vectorization process preserves semantic meaning and enables efficient similarity searches across large knowledge bases. As a result, queries can quickly locate the most relevant documents, significantly boosting both the speed and accuracy of responses.\n",
        "\n",
        "### 3.2 Retrieval and generation phase\n",
        "<img src=\"https://img.alicdn.com/imgextra/i3/O1CN01JJuO2e1TvrTOk9Dw7_!!6000000002445-2-tps-2987-938.png\" width=\"1000\">\n",
        "\n",
        "During the retrieval phase, the system searches the vector database for the most relevant document snippets based on the user’s query. These results are then combined with the original question and passed to the LLM, which generates a final, contextually informed answer. This allows the model to effectively respond to questions involving internal or proprietary knowledge without requiring full access to the underlying documents.\n",
        "\n",
        "In summary, applications based on the RAG structure avoid various issues caused by inputting entire reference documents as background information while extracting the most relevant parts through retrieval, thus improving the accuracy and relevance of the LLM output.\n",
        "\n",
        "## 📝 4. Summary\n",
        "In this section, we learned the following:\n",
        "\n",
        "- **How to use the LLM API**<br>\n",
        "    With hands-on code examples, you've seen how to interact with an LLM API and experienced its capabilities in answering questions and performing Q&A tasks.\n",
        "- **A preliminary understanding of how LLMs work**<br>\n",
        "    You now have a basic understanding of how LLMs interpret questions and generate responses. We also discussed some of their limitations — such as randomness and knowledge boundaries — and how techniques like prompt engineering and RAG can help overcome them.\n",
        "\n",
        "Beyond what we've shown here, LLMs are capable of much more than just answering questions. They can generate content, extract structured information, classify text, and analyze sentiment. Integrating RAG into your LLM applications also extends the range of knowledge they can leverage.\n",
        "\n",
        "In the next section, we’ll walk through how to build your own RAG-based applications.\n",
        "\n",
        "### Further reading\n",
        "- While going through this material, if you'd like to dive deeper into any of the topics, feel free to ask the LLM for additional explanations or learning resources.\n",
        "> For example, Qwen-Max includes an enable_search parameter that allows the model to enhance its answers by incorporating real-time web search results during response generation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "completion = client.chat.completions.create(\n",
        "    model=\"qwen-plus\",  # This example uses qwen-plus. You can replace it with other model names as needed. Model list: https://bailian.console.alibabacloud.com/?tab=doc#/doc/?type=model\n",
        "    messages=[\n",
        "        {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n",
        "        {\"role\": \"user\", \"content\": \"Number of gold medals won by Team America at the Paris Olympics\"},\n",
        "    ],\n",
        "    extra_body={\"enable_search\": True},\n",
        ")\n",
        "print(completion.choices[0].message.content)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The reasoning model [QwQ](https://help.aliyun.com/zh/model-studio/user-guide/qwq) of Alibaba Cloud has strong reasoning capabilities. The model will first output the thought process and then provide the response content.  \n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "reasoning_content = \"\"  # Define the complete reasoning process\n",
        "answer_content = \"\"     # Define the complete response\n",
        "is_answering = False   # Determine whether the reasoning process has ended and the response has started\n",
        "\n",
        "# Create a chat completion request\n",
        "completion = client.chat.completions.create(\n",
        "    model=\"qwq-32b\",  # Here, qwq-32b is used as an example; you can replace it with another model name as needed\n",
        "    messages=[\n",
        "        {\"role\": \"user\", \"content\": \"Which is larger, 9.9 or 9.11?\"}\n",
        "    ],\n",
        "    stream=True,\n",
        "    # Uncomment the following to return token usage in the last chunk\n",
        "    # stream_options={\n",
        "    #     \"include_usage\": True\n",
        "    # }\n",
        ")\n",
        "\n",
        "print(\"\\n\" + \"=\" * 20 + \" Reasoning Process \" + \"=\" * 20 + \"\\n\")\n",
        "\n",
        "for chunk in completion:\n",
        "    # If chunk.choices is empty, print usage\n",
        "    if not chunk.choices:\n",
        "        print(\"\\nUsage:\")\n",
        "        print(chunk.usage)\n",
        "    else:\n",
        "        delta = chunk.choices[0].delta\n",
        "        # Print the reasoning process\n",
        "        if hasattr(delta, 'reasoning_content') and delta.reasoning_content != None:\n",
        "            print(delta.reasoning_content, end='', flush=True)\n",
        "            reasoning_content += delta.reasoning_content\n",
        "        else:\n",
        "            # Start responding\n",
        "            if delta.content != \"\" and is_answering is False:\n",
        "                print(\"\\n\" + \"=\" * 20 + \" Complete Response \" + \"=\" * 20 + \"\\n\")\n",
        "                is_answering = True\n",
        "            # Print the response process\n",
        "            print(delta.content, end='', flush=True)\n",
        "            answer_content += delta.content"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "- If you are interested in multi-modal LLMs, you can refer to:\n",
        "\n",
        "    * [Visual Understanding](https://www.alibabacloud.com/help/en/model-studio/vision)\n",
        "    * [Omni-modal](https://www.alibabacloud.com/help/en/model-studio/qwen-omni)\n",
        "    * [AI Video Generation](https://www.alibabacloud.com/help/en/model-studio/use-video-generation)  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 🔥 Quiz\n",
        "\n",
        "### 🔍 Single choice question\n",
        "\n",
        "<details>\n",
        "<summary style=\"cursor: pointer; padding: 12px; border: 1px solid #dee2e6; border-radius: 6px;\">\n",
        "<b>What is the purpose of the following code snippet❓ (Select 1.)</b>\n",
        "<pre style=\"margin: 10px 0;\">\n",
        "os.environ[\"OPENAI_API_KEY\"] = \"your-api-key-here\" \n",
        "</pre>\n",
        "\n",
        "- A. Load the API key from disk  \n",
        "- B. Store the API key in memory  \n",
        "- C. Set the API key as an environment variable  \n",
        "- D. Create a new API key  \n",
        "\n",
        "**[Click to view answer]**\n",
        "</summary>\n",
        "\n",
        "<div style=\"margin-top: 10px; padding: 15px; border: 1px solid #dee2e6; border-radius: 0 0 6px 6px;\">\n",
        "\n",
        "✅ **Reference Answer: C**  \n",
        "📝 **Explanation**:  \n",
        "The code injects the API key into the current runtime environment's memory space using the operating system's environment variable interface.\n",
        "</div>\n",
        "</details>\n",
        "\n",
        "---\n",
        "\n",
        "### 📝 Case analysis question\n",
        "<details>\n",
        "<summary style=\"cursor: pointer; padding: 12px;  border: 1px solid #dee2e6; border-radius: 6px; font-color:#000;\">\n",
        "<b>Alex, while developing a writing assistant, encounters the following two scenarios. How should he solve these problems❓</b> \n",
        "\n",
        "- Scenario 🅰️ Lack of creativity in generated content: Every time he asks the model to write an article about \"the development of artificial intelligence,\" the generated content is very similar.  \n",
        "- Scenario 🅱️ Generated content deviates from the topic: When asking the model to write a technical document, the generated content often includes irrelevant information.  \n",
        "\n",
        "**Question:**  \n",
        "1. Based on the LLMs workflow learned in this section, what might be the causes of these issues in the two scenarios?  \n",
        "2. How should the temperature or top_p parameters be adjusted to address these problems?  \n",
        "\n",
        "**[Click to view answer]**\n",
        "</summary>\n",
        "\n",
        "\n",
        "<div style=\"margin-top: 10px;  padding: 15px;  border: 1px solid #dee2e6; border-radius: 0 0 6px 6px;\">\n",
        "\n",
        "#### 🎯 Solution for Scenario A\n",
        "**🔍 Cause Analysis**  \n",
        "The `temperature` value is too low (e.g., 0.3), causing the model to make single choices, resulting in a lack of diversity in the generated content.\n",
        "\n",
        "**⚙️ Parameter Adjustment**  \n",
        "\n",
        "\n",
        "\n",
        "```python\n",
        "temperature = 0.7~0.9  # Increase creativity\n",
        " top_p = 0.9            # Expand the range of word selection\n",
        "```\n",
        "\n",
        "#### 🎯 Solution for Scenario B\n",
        "**🔍 Cause Analysis**  \n",
        "`temperature` is too high (e.g., 1.2) or `top_p` is too large\n",
        "\n",
        "**⚙️ Parameter Adjustment**  \n",
        "\n",
        "\n",
        "\n",
        "```python\n",
        "temperature = 0.5~0.7  # Reduce randomness\n",
        " top_p = 0.7~0.8        # Focus on high probability words\n",
        "```\n",
        "\n",
        "🌟 **Parameter Tuning Tips**\n",
        "> It is recommended to adjust the parameters by ±0.2 each time and observe the effect changes through A/B testing.\n",
        "> If you need to balance Scenario A and Scenario B, it is recommended to use the combination: `temperature=0.6` + `top_p=0.8`</div>\n",
        "</details>  \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": []
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.12.10"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}
